Navigating Epidemics:
Accuracy and Utility of
Infectious Disease Models


Sebastian Funk (@sbfnk)
https://epiforecasts.io

27 February, 2024
MPI-DS Colloquium

Acknowledgements

EpiForecasts group (https://epiforecasts.io):
Akira Endo, Alexis Robert, Ciara McCarthy, Hannah Choi,
Joel Hellewell, James Azam, James Munday, Kath Sherratt,
Liza Hadley, Manuel Stapper, Nikos Bosse, Sam Abbott,
Sophie Meakin, Toshiaki Asakura


Collaborators at LSHTM and elsewhere.

Models are a tool to combine data (what we know) with assumptions and theory (what we think) to learn about what we don’t know.

When data is abundant, models and analytics can generate insight without many additional assumptions.

When data is sparse (e.g. early in an outbreak), modellers need to make more assumptions to generate insights.

January 2020: Can COVID-19 be controlled by contact tracing?

Hellewell et al., medRxiv, 2020

Probability of control depends on intensity of transmission and contact tracing effort.

Hellewell et al., medRxiv, 2020

“We illustrate the potential impact that flawed model inferences can have on public health policy with the model described […] by Joel Hellewell and colleagues, which is part of the scientific evidence informing the UK Government’s response to COVID-19.”

Gudrasani & Ziauddeen, Lancet Glob Health, 2020

“All models are wrong, but some are useful”

– George Box

“All models are wrong, but some are useful”

– George Box

wrong: how wrong?
some: which ones?

The future as a (particular) data gap

The future as a (particular) data gap

Mechanistic models support causal understanding, but predictions can have value in their own right

Desai et al., Health Secur, 2019
Keeling et al., Stat Meth Med Res, 2021
Cramer et al., Scientific Data, 2021

Short-term forecasts can inform decision making

Finger et al., BMC Medicine, 2019

Anticipate healthcare demand

Short-term forecasts can inform decision making

Design interventions

Short-term forecasts can inform decision making

Plan clinical trials

Forecasting Ebola

Ebola in West Africa, 2013-16

“We were losing ourselves in details […] all we needed to know is, are the number of cases rising, falling, or levelling off?”

– Hans Rosling, Liberia, 2014

Forecasts can be assessed/validated

Forecasting paradigm

“maximise sharpness subject to calibration”

Gneiting, Balabdaoui & Raftery, J R Stat Soc B, 2007

Ebola: how wrong were our models?

Funk et al., PLOS Comp Biol, 2019

Ebola forecasts could be trusted for up to 2 weeks

Our Ebola forecasts could be trusted for up to 2 weeks

Forecasting COVID-19

Forecasting via the renewal equation

\begin{align} \textrm{New infections}~I(t) & = R_t \sum_{\tau} g_{\tau} I_{t-\tau}\\ \textrm{Reproduction number}~R(t) & = R_{t-1} \exp(\textrm{GP})\\ \textrm{Delayed reporting}~D(t) & = \sum_\tau \xi_\tau I_{t-\tau} \\ \textrm{Observations}~C(t) & = \mathrm{NegBin}(D_t \omega_{(t~\textrm{mod}~7)}, \phi) \end{align}

https://epiforecasts.io/EpiNow2/

Global COVID case forecasts via the renewal equation

Abbott et al., Wellcome Open Res, 2020
Gostic et al., PLoS Comp Biol, 2021
https://epiforecasts.io/posts/2022-03-25-rt-reflections

Forecasting to inform policy in the UK

Funk et al., medRxiv, 2021
Medley, Adv Biol Regul, 2022
Whitty et al., 2023

European COVID-19 Forecast Hub

Reich et al., Am J Public Health, 2022
https://covid19forecasthub.org
https://covid19forecasthub.eu

How good were COVID forecasts?

We can compare forecasts using
proper scoring rules \[\mathrm{CRPS}(F, x) = \mathbb{E}|X-x| - \frac{1}{2}\mathbb{E}|X-X'|\]

Gneiting and Raftery, J R Statist Soc B, 2007

Median ensemble outperforms individual models

Sherratt et al., medRxiv, 2022

We can compare forecasts using proper scoring rules \[\mathrm{CRPS}(F, x) = \mathbb{E}|X-x| - \frac{1}{2}\mathbb{E}|X-X'|\] but these only tell us about relative quality of forecasts

Absolute quality of forecasts #1: baseline models

Absolute quality of forecasts #2: calibration

Sherratt et al., medRxiv, 2022

What limits predictive ability?

  1. Unpredictable human behaviour?
  2. Unpredictable pathogen biology?
  3. Bad models?

Unpredictable human behavoiur?

Gimma et al., PLOS Medicine, 2022

Observed behaviour as predictor: improvement of forecasts, but only once age-specific reporting is taken into account.

Munday et al., PLOS Comp Biol, 2023

Unpredictable biology?

Lythgoe et al., Proc Roy Soc B, 2023

Variants as predictor: improvements of forecasts during transitions

https://github.com/epiforecasts/forecast.vocs
https://github.com/jbracher/branching_process_delta

Bad models? Human vs. machine.

Bosse et al., PLOS Comp Biol, 2022

Humans better than models at predicting cases, but not deaths

Bosse et al., PLOS Comp Biol, 2022

Amongst models, ones that focus on a single country tended to do better

Sherratt et al., work in progress

Inherent limits?

Scarpino & Petri, Nat Comm, 2019
Gamarnik & Ma, medRxiv, 2024

What can we conclude for the next pandemic?

Summary

  • Covid-19 forecasts have been relatively poor further than one or two generations ahead
  • Ensembles perform best, but can be difficult to interpret

Open questions:

  • Can predictive performance be improved?
  • Are we measuring predictive performance in the right way?

Alternative ways of measuring predictive performance change the ranking of models.

Gerding et al., arXiv, 2024// Bosse et al., PLOS Comp Biol, 2023

Forecasting and nowcasting remain relevant

UKHSA, 2022
Overton et al., PLOS Comp Biol, 2023

New initiatives

European respiratory hub

https://respicast.ecdc.europa.eu/

Evaluating conditional forecasts (“scenarios”)

Howerton et al., Nat Comm, 2023

We need collaborative efforts, using standardised datasets to compare methods and generating sustainable tools

“We were losing ourselves in details […] all we needed to know is, are the number of cases rising, falling or levelling off?”

– Hans Rosling, Liberia, 2014

Slides at

https://epiforecasts.io/slides/mpi_goettingen_20240227.html